This repository was archived by the owner on Sep 11, 2020. It is now read-only.
plumbing: use sliding window in delta calculations, like git CL #565
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 25, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
Codecov Report
@@ Coverage Diff @@
## master #565 +/- ##
=========================================
- Coverage 77.98% 77.38% -0.6%
=========================================
Files 129 129
Lines 9845 9845
=========================================
- Hits 7678 7619 -59
- Misses 1328 1400 +72
+ Partials 839 826 -13
Continue to review full report at Codecov.
|
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 26, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 26, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 26, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
|
Any chance to cover this with a test? |
erizocosmico
approved these changes
Aug 27, 2017
This sets a default sliding window of 10 for the delta calculation, just like git CL: https://git-scm.com/docs/git-pack-objects#git-pack-objects---windowltngt For a big-ish repo with 35K objects (17K commits), this reduced the time for calling `deltaSelection.walk` during a push from more than 14 minutes to about a minute.
strib
force-pushed
the
delta-sliding-window
branch
from
9801511 to
cdddb7a
Compare
|
Thanks @mcuadros and @erizocosmico, added a simple test. Ready for merge, I think. |
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 28, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 28, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
# for free
to subscribe to this conversation on GitHub.
Already have an account?
#.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This sets a default sliding window of 10 for the delta calculation, just like git command line:
https://git-scm.com/docs/git-pack-objects#git-pack-objects---windowltngt
For a big-ish repo with 35K objects (17K commits), this reduced the time for calling
deltaSelection.walkduring a push from more than 14 minutes to about a minute.